Team Nova DataFest22's Datathon ML Submission¶

Help a Financial Industry predict who is likely to complete an E-process application

Project Overview¶

Background¶

BorrowMe is a peer-to-peer lending and renting platform that connects individuals who own underused items with people who need them temporarily. The platform benefits everyone involved. Lenders make money, borrowers save money, and the Earth's toxic load is reduced—one platform to help them all.

My team just got hired as a Data Science team at Borrowme, a fintech playing in the lending space. The CEO tasks us with helping to revamp the lending process due to complaints from customers on the time the process takes

Project Task¶

The IT team has provided historical information containing the information of applicants and the status of their loan application process. You are expected to automate the process by building a machine learning model to predict the outcome of the lending process if the credit facility process was completed via e-process, that is E-Signed or not.

Importing the Packages¶

In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
from lightgbm import LGBMClassifier
import optuna
from sklearn.metrics import f1_score, roc_auc_score
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.feature_selection import mutual_info_classif
from sklearn.model_selection import StratifiedKFold, RandomizedSearchCV, cross_val_score, GridSearchCV
from sklearn.preprocessing import KBinsDiscretizer

warnings.filterwarnings('ignore')
pd.set_option('display.max_columns',None)

Data Gathering¶

The datasets provided for this project are imported using pandas and loaded into a dataframe

In [2]:
# to read the train.csv dataset into a dataframe 
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')

# to see the overview of the table
train.head()
Out[2]:
Entry_id age pay_schedule home_owner income months_employed years_employed current_address_year personal_account_m personal_account_y has_debt amount_requested risk_score risk_score_2 risk_score_3 risk_score_4 risk_score_5 ext_quality_score ext_quality_score_2 inquiries_last_month e_signed
0 7629673 40 bi-weekly 1 3135 0 3 3 6 2 1 550 36200 0.737398 0.903517 0.487712 0.515977 0.580918 0.380918 10 1
1 5335819 33 semi-monthly 0 3590 0 5 2 2 8 1 1100 53850 0.617361 0.857560 0.613487 0.665523 0.744634 0.744634 12 0
2 8492423 21 weekly 0 2303 0 5 8 2 7 1 600 74850 0.677109 0.758765 0.495609 0.664762 0.592556 0.492556 6 1
3 7948313 26 bi-weekly 0 2795 0 4 4 1 6 1 800 50800 0.738055 0.873204 0.666437 0.700392 0.584130 0.684130 14 1
4 4297036 43 bi-weekly 0 5000 0 2 1 1 2 1 1100 69100 0.798303 0.841747 0.401971 0.568787 0.525905 0.725905 5 1

Columns¶

  • Entry_id : User Identifier - client
  • age: Age of user
  • pay_schedule: How often applicants get paid
  • home_owner: 0:- Rented home 1:- Owner
  • income: Applicant income
  • months_employed: How many months has he been doing the job
  • years_employed: How many years he been doing job
  • current_address_year: How many years a person stayed at the current address
  • personal_account_m: How many months that person had a personal account
  • personal_account_y: How many years has that person had a personal account
  • has_debt: If the individual has debt
  • amount_requested: Loan amount requested
  • risk_score_n (1,2,3,4,5):- Various scores are defined by regulations and internal processes to classify individual risks
  • ext_quality_score_n (1 & 2): External quality scores defined for each individual
  • inquiries_last_month: Number of times an individual made inquiries in the previous month.
  • e_signed: If the application was completed as an e-signing process or not

Data Assessing and Cleaning¶

In this section, the dataset is assessed for Quality and Tidiness issues and cleaned to obtain a clean data

In [3]:
# to get the info of the table
train.info() 
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 12516 entries, 0 to 12515
Data columns (total 21 columns):
 #   Column                Non-Null Count  Dtype  
---  ------                --------------  -----  
 0   Entry_id              12516 non-null  int64  
 1   age                   12516 non-null  int64  
 2   pay_schedule          12516 non-null  object 
 3   home_owner            12516 non-null  int64  
 4   income                12516 non-null  int64  
 5   months_employed       12516 non-null  int64  
 6   years_employed        12516 non-null  int64  
 7   current_address_year  12516 non-null  int64  
 8   personal_account_m    12516 non-null  int64  
 9   personal_account_y    12516 non-null  int64  
 10  has_debt              12516 non-null  int64  
 11  amount_requested      12516 non-null  int64  
 12  risk_score            12516 non-null  int64  
 13  risk_score_2          12516 non-null  float64
 14  risk_score_3          12516 non-null  float64
 15  risk_score_4          12516 non-null  float64
 16  risk_score_5          12516 non-null  float64
 17  ext_quality_score     12516 non-null  float64
 18  ext_quality_score_2   12516 non-null  float64
 19  inquiries_last_month  12516 non-null  int64  
 20  e_signed              12516 non-null  int64  
dtypes: float64(6), int64(14), object(1)
memory usage: 2.0+ MB
In [4]:
# to get the shape of the table
train.shape
Out[4]:
(12516, 21)

From the above result, There are 12,516 rows and 21 columns

In [5]:
# to check for the presence of null values
train.isnull().sum()
Out[5]:
Entry_id                0
age                     0
pay_schedule            0
home_owner              0
income                  0
months_employed         0
years_employed          0
current_address_year    0
personal_account_m      0
personal_account_y      0
has_debt                0
amount_requested        0
risk_score              0
risk_score_2            0
risk_score_3            0
risk_score_4            0
risk_score_5            0
ext_quality_score       0
ext_quality_score_2     0
inquiries_last_month    0
e_signed                0
dtype: int64

From the above analysis, there are no null values presence in the train table.

In [6]:
# to check for duplicate rows
sum(train.duplicated())
Out[6]:
0

There are no duplicate records in the train data. To check if there are duplicate records in the entry_id column

In [7]:
# to check for duplicated records in the entry_id column
train['Entry_id'].duplicated().sum()
Out[7]:
6

There have been 6 observed duplicates in the "Entry_id" column. Further investigation would be done

In [8]:
# to print out the rows where the ID is in the IDs of duplicated rows
ids = train['Entry_id']
duplicate_id = ids[ids.duplicated()]
train[ids.isin(duplicate_id)].sort_values('Entry_id')
Out[8]:
Entry_id age pay_schedule home_owner income months_employed years_employed current_address_year personal_account_m personal_account_y has_debt amount_requested risk_score risk_score_2 risk_score_3 risk_score_4 risk_score_5 ext_quality_score ext_quality_score_2 inquiries_last_month e_signed
2488 3903643 41 bi-weekly 0 3035 0 5 0 2 6 0 700 56550 0.574753 0.843991 0.241663 0.516921 0.715925 0.715925 15 1
9804 3903643 38 bi-weekly 1 2620 0 2 0 6 4 1 400 67050 0.740419 0.939486 0.684477 0.778170 0.629805 0.729805 9 1
7213 5659209 42 bi-weekly 1 2140 0 2 9 6 4 1 400 56700 0.781691 0.920329 0.481009 0.594963 0.736846 0.536846 6 1
7288 5659209 46 bi-weekly 1 7092 0 3 1 2 3 0 700 64350 0.679462 0.757643 0.550134 0.391500 0.487792 0.587792 6 1
5361 5896278 31 weekly 0 1635 0 6 5 2 2 1 400 73200 0.797840 0.849569 0.687969 0.741433 0.696454 0.696454 6 0
11829 5896278 30 bi-weekly 0 3795 0 2 3 1 3 1 800 50650 0.760728 0.850893 0.573459 0.606560 0.564857 0.564857 4 1
4419 5946902 43 semi-monthly 1 4725 3 6 6 5 8 1 1000 59250 0.724398 0.944054 0.444145 0.582691 0.698231 0.698231 3 1
12124 5946902 41 semi-monthly 1 4295 2 2 1 3 2 1 1200 54100 0.681107 0.860848 0.525299 0.689527 0.421167 0.621167 9 0
2626 7450902 57 bi-weekly 0 5295 0 6 4 2 6 1 1700 91500 0.648824 0.789179 0.640210 0.653759 0.694329 0.594329 5 0
9209 7450902 35 bi-weekly 0 4095 6 2 0 1 3 0 700 57850 0.780735 0.929274 0.625784 0.792078 0.550449 0.550449 5 0
1487 8156839 37 bi-weekly 0 3020 0 1 7 4 4 1 600 53550 0.728198 0.881015 0.647649 0.742549 0.594076 0.494076 5 1
11067 8156839 48 bi-weekly 0 3205 0 2 0 5 3 1 500 53650 0.623579 0.801490 0.621682 0.654174 0.395154 0.495154 4 1

The rows with duplicate "Entry_id" were observed and we noticed that thier fields were different so We decided not to drop these rows and thier values might influence our analysis.

.

In [9]:
# to check for the percentage of unique records
train.nunique()/len(train)
Out[9]:
Entry_id                0.999521
age                     0.005753
pay_schedule            0.000320
home_owner              0.000160
income                  0.155002
months_employed         0.000959
years_employed          0.001358
current_address_year    0.001039
personal_account_m      0.000959
personal_account_y      0.001198
has_debt                0.000160
amount_requested        0.007430
risk_score              0.109460
risk_score_2            0.983301
risk_score_3            0.283477
risk_score_4            0.989773
risk_score_5            0.985858
ext_quality_score       0.983381
ext_quality_score_2     0.983701
inquiries_last_month    0.002397
e_signed                0.000160
dtype: float64

Explorative Data Analysis:¶

In [10]:
# to check for the descriptive statistics of the numerical columns
train.describe()
Out[10]:
Entry_id age home_owner income months_employed years_employed current_address_year personal_account_m personal_account_y has_debt amount_requested risk_score risk_score_2 risk_score_3 risk_score_4 risk_score_5 ext_quality_score ext_quality_score_2 inquiries_last_month e_signed
count 1.251600e+04 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000 12516.000000
mean 5.599485e+06 43.026766 0.425056 3654.110658 1.191355 3.532998 3.593161 3.402205 3.508469 0.795462 950.251518 61133.401246 0.690665 0.877954 0.582848 0.718456 0.621944 0.621912 6.494088 0.540588
std 2.563354e+06 11.865891 0.494371 1508.072163 2.404529 2.265293 2.749211 2.206649 1.959969 0.403380 706.915146 15396.809340 0.090294 0.054197 0.125260 0.121103 0.139876 0.139623 3.731870 0.498370
min 1.111398e+06 18.000000 0.000000 905.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 350.000000 2100.000000 0.023258 0.451371 0.016724 0.153367 0.010184 0.006622 1.000000 0.000000
25% 3.376885e+06 34.000000 0.000000 2575.000000 0.000000 2.000000 2.000000 2.000000 2.000000 1.000000 600.000000 49450.000000 0.641221 0.850893 0.499887 0.635005 0.519794 0.519662 4.000000 0.000000
50% 5.608376e+06 42.000000 0.000000 3255.000000 0.000000 3.000000 3.000000 2.000000 3.000000 1.000000 700.000000 61200.000000 0.699621 0.880993 0.588202 0.725717 0.624081 0.623781 6.000000 1.000000
75% 7.803325e+06 51.000000 1.000000 4661.250000 1.000000 5.000000 5.000000 5.000000 4.000000 1.000000 1100.000000 72750.000000 0.752053 0.911485 0.672431 0.806576 0.729609 0.728389 8.000000 1.000000
max 9.999874e+06 96.000000 1.000000 9985.000000 11.000000 16.000000 12.000000 11.000000 14.000000 1.000000 10200.000000 99750.000000 0.999997 0.999012 0.978932 0.993363 0.970249 0.966953 30.000000 1.000000
In [11]:
# to visualize the distribution of the data

train.hist(figsize = (40, 60), color = 'turquoise');

.

Percentage of clients that are e-signed¶

This analysis is to investigate the percentage of the clients who got E_signed and not

In [12]:
# to count the number of e-signed clients 
sorted_counts = train['e_signed'].value_counts()
print(sorted_counts)

# to represent it in a pie plot
plt.figure(figsize = (10,6))
labels = ['E-signed', 'Not E-signed']
plt.pie(sorted_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of E-signed Clients', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12)
plt.show();
1    6766
0    5750
Name: e_signed, dtype: int64

From this insight generated, it shows that more than 50% of the clients are E-signed

Percentage of Clients that are E-signed and in debt¶

This Analysis was done to observe the percentage of clients that are E-signed and its relationship to their debt status

In [13]:
# first lets check the number of clients in debt
debt_counts = train['has_debt'].value_counts()
print(debt_counts)

# to represent it in a pie plot
plt.figure(figsize = (10,6))
labels = ['Have Debt', 'No Debt']
plt.pie(debt_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of Clients who Have Debt', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12)
plt.show();
1    9956
0    2560
Name: has_debt, dtype: int64
In [14]:
# to check have_debt along with e-signed to explore relationship
plt.figure(figsize = [16,5])

# left plot 
# this is for clients who have debt
plt.subplot(1,2,1)
debt = train[train['has_debt'] == 1]

# to count the number of e-signed clients
debt_esign_counts = debt['e_signed'].value_counts()
print(debt_esign_counts)

# to represent it in a pie plot
labels = ['E-signed', 'Not E-signed']
plt.pie(debt_esign_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of E-signed Clients who has Debt', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12)

# right plot
# this is for clients who do not have debt
plt.subplot(1,2,2)
not_debt = train[train['has_debt'] == 0]

# to count the number of e-signed clients
not_debt_esign_counts = not_debt['e_signed'].value_counts()
print(not_debt_esign_counts)

# to represent it in a pie plot
labels = ['E-signed', 'Not E-signed']
plt.pie(not_debt_esign_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of E-signed Clients who has no Debt', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12);
1    5465
0    4491
Name: e_signed, dtype: int64
1    1301
0    1259
Name: e_signed, dtype: int64
  • From the Analysis above, 79.55% (9,956) of the clients have debt while the remaining 20.45% (2,560) of the clients do not have debt.

  • Out of the 9,956 clients who have debt, 54.89% (5,465) of them are E-signed while the remaining 45.11% (4,491) of the clients are not E-signed.

  • Out of the 2,560 clients who have no debt, 50.82% (1,301) of them are E-signed while the remaining 49.18% (1,259) of the clients are not E-signed.

Percentage of Clients that are E-signed and house owners¶

This analysis aims to observe the percentage of clients that are house owners and observe the relationship with E-signed

In [15]:
# first lets check the number of clients in debt
house_owner = train['home_owner'].value_counts()
print(house_owner)

# to represent it in a pie plot
plt.figure(figsize = (10,6))
labels = ['Not Home Owner', 'Home Owner']
plt.pie(house_owner, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['lightgray', 'turquoise'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of Clients who are Home Owners', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12)
plt.show();
0    7196
1    5320
Name: home_owner, dtype: int64
In [16]:
# to check House Owner along with e-signed to explore relationship
plt.figure(figsize = [16,5])

# left plot 
# this is for clients who have debt
plt.subplot(1,2,1)
home = train[train['home_owner'] == 1]

# to count the number of e-signed clients
home_esign_counts = home['e_signed'].value_counts()
print(home_esign_counts)

# to represent it in a pie plot
labels = ['E-signed', 'Not E-signed']
plt.pie(home_esign_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of E-signed Clients who are Home Owners', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12)

# right plot
# this is for clients who do not have debt
plt.subplot(1,2,2)
not_home = train[train['home_owner'] == 0]

# to count the number of e-signed clients
not_home_esign_counts = not_home['e_signed'].value_counts()
print(not_home_esign_counts)

# to represent it in a pie plot
labels = ['E-signed', 'Not E-signed']
plt.pie(not_home_esign_counts, labels = labels, radius = 1.8, autopct = '%0.2f%%', colors = ['turquoise', 'lightgray'],
        textprops = {'fontsize' : 12}, startangle= 90, counterclock= False, wedgeprops= {'width' : 1.0});
plt.axis('equal')
plt.title('Percentage of E-signed Clients who are not Home Owners', fontsize = 14 )
plt.legend(loc = 'upper right', fontsize = 12);
1    2707
0    2613
Name: e_signed, dtype: int64
1    4059
0    3137
Name: e_signed, dtype: int64
  • From the Analysis above, 42.51% (5,320) of the clients are Home Owners while the remaining 57.49% (7,196) of the clients are not Home Owners.

  • Out of the 5,320 clients who are Home Owners, 50.88% (2,707) of them are E-signed while the remaining 49.12% (2,613) of the clients are not E-signed.

  • Out of the 7,196 clients who are not Home Owners, 56.41% (4,059) of them are E-signed while the remaining 43.59% (3,137) of the clients are not E-signed.

Age Distribution of the Clients with respect to E-signed¶

This analysis aims to observe the age distribution of the clients and its relationship to E-signed

In [17]:
# first let us find the age distribution of the clients 
plt.figure(figsize= [20,10])
sns.set_style('white')

# left plot showing the histogram distribution of age
plt.subplot(1,2,1)
bins = np.arange(0, train['age'].max() + 5, 5)
sns.distplot(train['age'], kde = False, bins = bins, color= 'turquoise');
plt.title("Histogram Showing the Distribuion of Clients Age", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("Age", fontsize = 14)

# right plot showing the kernel density plot of Client's Age
plt.subplot(1,2,2)
bins = np.arange(0, train['age'].max() + 5, 5)
sns.distplot(train['age'], hist = False, bins = bins, color= 'turquoise');
plt.title("Kernel Density Plot of CLients Age", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("Age", fontsize = 14);
In [18]:
# let us observe the age distribution of the clients with respect to e-signed 
plt.figure(figsize= [20,10])
sns.set(style = 'dark')

# left plot showing the histogram distribution of age
plt.subplot(1,2,1)
esign = train[train['e_signed'] == 1]
bins = np.arange(0, esign['age'].max() + 5, 5)
sns.distplot(esign['age'], bins = bins, color= 'turquoise');
plt.title("Distribuion of Clients Age that are E-signed")

# right plot showing the kernel density plot of Client's Age
plt.subplot(1,2,2)
not_esign = train[train['e_signed'] == 0]
bins = np.arange(0, not_esign['age'].max() + 5, 5)
sns.distplot(not_esign['age'], bins = bins, color= 'turquoise');
plt.title("Distribuion of Clients Age that are not E-signed");

From the Analysis above, ages 40-45 appears to be the most age gaps among the clients. This distribution does not change when observed alongside the e-signed column. The age distribution does not have an relationship with the E-signed process

Examining the Number of clients and their pay schedule¶

This analysis aims to observe the pay schedule of the clients then look for its relationship with e-signed

In [19]:
# observing the overview of the table
train.head()
Out[19]:
Entry_id age pay_schedule home_owner income months_employed years_employed current_address_year personal_account_m personal_account_y has_debt amount_requested risk_score risk_score_2 risk_score_3 risk_score_4 risk_score_5 ext_quality_score ext_quality_score_2 inquiries_last_month e_signed
0 7629673 40 bi-weekly 1 3135 0 3 3 6 2 1 550 36200 0.737398 0.903517 0.487712 0.515977 0.580918 0.380918 10 1
1 5335819 33 semi-monthly 0 3590 0 5 2 2 8 1 1100 53850 0.617361 0.857560 0.613487 0.665523 0.744634 0.744634 12 0
2 8492423 21 weekly 0 2303 0 5 8 2 7 1 600 74850 0.677109 0.758765 0.495609 0.664762 0.592556 0.492556 6 1
3 7948313 26 bi-weekly 0 2795 0 4 4 1 6 1 800 50800 0.738055 0.873204 0.666437 0.700392 0.584130 0.684130 14 1
4 4297036 43 bi-weekly 0 5000 0 2 1 1 2 1 1100 69100 0.798303 0.841747 0.401971 0.568787 0.525905 0.725905 5 1
In [20]:
# to plot a countplot 
sns.set_style('white')
counts = train['pay_schedule'].value_counts()
counts_order = counts.index
sns.countplot(data = train, x = 'pay_schedule', color= 'turquoise', order= counts_order);
In [21]:
# to examine the countplot based on E-signed 
counts = train['pay_schedule'].value_counts()
counts_order = counts.index
sns.catplot(data = train, x = 'pay_schedule', kind = 'count', color= 'turquoise', order= counts_order, col = 'e_signed');
plt.show();
In [22]:
# for clarity purpose, a heatmap is plotted to show the counts 
plt.figure(figsize= (12,8))
pay_counts = train.groupby(['pay_schedule','e_signed']).size()
pay_counts = pay_counts.reset_index(name= 'count')
pay_counts = pay_counts.pivot(columns = 'pay_schedule', index = 'e_signed', values= 'count')
sns.heatmap(pay_counts, annot= True, fmt= 'd', cmap= 'PuBuGn')
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('E_Signed', fontsize = 14)
plt.xlabel('Pay Schedule', fontsize = 14)
plt.title("HeatMap showing Count of Pay Schedules for clients for E_signed Loans", fontsize= 16);

From the analysis above, Mosgt of our clients earn biweekly followed by Monthly and weekly being the least.

After observing the pay schedule with e-signed column, a relationship was not spotted.

Examining the distribution of amount requested¶

In [23]:
train['amount_requested'].describe()
Out[23]:
count    12516.000000
mean       950.251518
std        706.915146
min        350.000000
25%        600.000000
50%        700.000000
75%       1100.000000
max      10200.000000
Name: amount_requested, dtype: float64
In [24]:
# first let us find the distribution of the amount requested by clients 
plt.figure(figsize= [20,10])

# left plot showing the histogram distribution of amount requested by clients
plt.subplot(1,2,1)
bins = np.arange(0, train['amount_requested'].max() + 500, 500)
sns.distplot(train['amount_requested'], kde = False, bins = bins, color= 'turquoise');
plt.title("Histogram Showing the Distribuion of Amount Requested", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("Amount Requested", fontsize = 14)

# right plot showing the kernel density plot of amount requested by clients
plt.subplot(1,2,2)
bins = np.arange(0, train['amount_requested'].max() + 500, 500)
sns.distplot(train['amount_requested'], hist = False, bins = bins, color= 'turquoise');
plt.title("Kernel Density Plot of Amount Requested", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("Amount Requested", fontsize = 14);
In [25]:
# to examine the relationship between Amount requested and e-signed
plt.figure(figsize= (12,8))
sns.violinplot(data= train, x = 'e_signed', y= 'amount_requested', color= 'turquoise')
plt.yticks([0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000], fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Amount Requested', fontsize = 14)
plt.xlabel('E_signed', fontsize = 14)
plt.title("Distribution of Amount Requested with respect to E_signed", fontsize= 16);

After observing the distribution of the amount requested, we figured out that majority of the amount requested fell within 500 - 1500. The distribution was also observed with respect to the E_signed status and we observed that, clients who are E-signed did not request for amounts greater than 4500 while clients who are not E_signed have requested for loan amount up to 10000

To Examine the relationship between income and amount requested¶

In [26]:
# to plot a scatter plot income and amount requested
plt.figure(figsize= (12,8))
sns.regplot(x = 'income', y = 'amount_requested', data = train, color = 'turquoise', truncate= False)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Amount Requested', fontsize = 14)
plt.xlabel("Client's Income", fontsize = 14)
plt.title("Relationship Between Amount Requested and Client Income", fontsize= 16)
plt.show();
In [27]:
plt.figure(figsize= (12,8))
hue_colors = {0: 'lightgray', 1: 'turquoise'}
sns.scatterplot(x = 'income', y = 'amount_requested', hue= 'e_signed', data = train, palette= hue_colors)

# setting an amount requested mark at 4200
plt.axhline(4200, ls= '--', color= 'black')
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Amount Requested', fontsize = 14)
plt.xlabel("Client's Income", fontsize = 14)
plt.title("Relationship Between Amount Requested and Client Income", fontsize= 16)
plt.show();
In [28]:
# to seperate the scatterplot based on e-signed
plt.figure(figsize=[20, 12])
g = sns.relplot(x = 'income', y = 'amount_requested', data = train, kind = 'scatter', col = 'e_signed', color = 'turquoise')
axes = g.axes.flatten()

for i, ax in enumerate(axes):
    ax.axhline(4200, ls = '--', c = 'black')
plt.show();
<Figure size 1440x864 with 0 Axes>
In [29]:
# to seperate the scatterplot based on e-signed and has debt

g = sns.relplot(x = 'income', y = 'amount_requested', data = train, kind = 'scatter', col = 'e_signed', row = 'has_debt',
            color = 'turquoise')
axes = g.axes.flatten()

for i, ax in enumerate(axes):
    ax.axhline(4200, ls = '--', c = 'black')

plt.show();

From the analysis above, we deduced that clients who requested for amounts greater than 4200 got rejected for their loan approval (E-signed) irrespective of the clients income.

On further investigation, we figured out the clients who requested above the certain benchmark of 4200 are clients who are not E_signed and who are in debt.

Examining the relationship between risk score and external quality score and Income¶

THis analysis aims to observe the relationship between risk score, external quality score and income

In [30]:
# to calculate the mean external quality score
train['mean_quality'] = train[['ext_quality_score', 'ext_quality_score_2']].mean(axis = 1)

# to calculate the mean risk score using risk score 2, 3, 4 & 5
col = train.loc[:, "risk_score_2":"risk_score_5"]
train['mean_risk_score'] = col.mean(axis= 1)

# to check if the columns have been added
train.head();
In [31]:
# to check the relationship between mean_risk_score, ext_quality_score and income
column = train[['mean_risk_score', 'mean_quality', 'income']]
print(column.head())

# using pairplot to observe relationships
sns.pairplot(column, height= 4);
   mean_risk_score  mean_quality  income
0         0.661151      0.480918    3135
1         0.688483      0.744634    3590
2         0.649061      0.542556    2303
3         0.744522      0.634130    2795
4         0.652702      0.625905    5000
In [32]:
# to check the correlation between mean quality score and mean risk score
train['mean_risk_score'].corr(train['mean_quality'])
Out[32]:
0.4190188215401355

A positive correlation of 0.42 was observed between the mean quality score and the mean risk score

To observe the relationship with respect to E-signed

In [33]:
sns.relplot(data= train, x = 'mean_risk_score', y = 'mean_quality', col= 'e_signed', kind= 'scatter', color = 'turquoise')
plt.show();
In [34]:
# to check the correlation
esign = train[train['e_signed'] == 1]
not_esign = train[train['e_signed'] == 0]

esign_cor = esign['mean_risk_score'].corr(esign['mean_quality'])
not_esign_cor = not_esign['mean_risk_score'].corr(not_esign['mean_quality'])

print(esign_cor)
print(not_esign_cor)
0.4270416332733701
0.40958524675311725

From this analysis, there is a stronger correlation between mean quality and mean risk score for e_signed clients compared to clients who are not e_signed

To Examine the relationship between number of enquiries last month and E-signed¶

THis analysis aims to study the relationship between the number of enquiries last month and the E_signed status

In [35]:
# first let us find the distribution of the Number of Inquiries Last Month
plt.figure(figsize= [20,10])

# left plot showing the histogram distribution of Number of Inquiries Last Month
plt.subplot(1,2,1)
bins = np.arange(0, train['inquiries_last_month'].max() + 2, 2)
sns.distplot(train['inquiries_last_month'], kde = False, bins = bins, color= 'turquoise');
plt.title("Histogram Showing the Distribuion of the Number of Inquiries Last Month", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("inquiries_last_month", fontsize = 14)

# right plot showing the kernel density plot of Number of Inquiries Last Month
plt.subplot(1,2,2)
bins = np.arange(0, train['inquiries_last_month'].max() + 4, 4)
sns.distplot(train['inquiries_last_month'], hist = False, bins = bins, color= 'turquoise');
plt.title("Kernel Density Plot of the Number of Inquiries Last Month", fontsize= 16)
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.xlabel("inquiries_last_month", fontsize = 14);
In [36]:
# to examine the relationship between Amount requested and e-signed
plt.figure(figsize= (12,8))
sns.violinplot(data= train, x = 'e_signed', y= 'inquiries_last_month', color= 'turquoise')
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Number of Inquiries last Month', fontsize = 14)
plt.xlabel('E_signed', fontsize = 14)
plt.title("Distribution of Number of Inquiries last Month with respect to E_signed", fontsize= 16);
In [37]:
# to examine the relationship further using has_debt variable
plt.figure(figsize= [20,25])

debts = train[train['has_debt'] == 1]
nodebts = train[train['has_debt'] == 0]

# left plot is for clients who have debt
plt.subplot(2,1,1)
sns.violinplot(data= debts, x = 'e_signed', y= 'inquiries_last_month', color= 'turquoise')
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Number of Inquiries last Month', fontsize = 14)
plt.xlabel('E_signed', fontsize = 14)
plt.title("Distribution of Number of Inquiries last Month For Clients with Debt with respect to E_signed", fontsize= 16);

# right plot is for clients who do not have debt
plt.subplot(2,1,2)
sns.violinplot(data= nodebts, x = 'e_signed', y= 'inquiries_last_month', color= 'turquoise')
plt.yticks(fontsize = 14)
plt.xticks(fontsize = 14)
plt.ylabel('Number of Inquiries last Month', fontsize = 14)
plt.xlabel('E_signed', fontsize = 14)
plt.title("Distribution of Number of Inquiries last Month For Clients without Debt with respect to E_signed", fontsize= 16);

From the Analysis above, The distribution of the number of enquiries per month had its peak around 4-8 inquiries.

On observing the relationship between the distribution and the E_signed column, there was no observed relationship between the two variables.

Conclusion:¶

The features included in the dataset holds relationships that can be inferred to the target variable, hence they'd be preprocessed and sequentially used to develop a model for an optimal solution


Model Development:¶

In [38]:
train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
In [39]:
# computing 'class_weight'
from sklearn.utils import class_weight
class_weight = dict(zip(np.unique(train['e_signed']), class_weight.compute_class_weight(class_weight = 'balanced',classes = np.unique(train['e_signed']),
                                                                                        y = train['e_signed']))) 
class_weight
Out[39]:
{0: 1.0883478260869566, 1: 0.9249187112030742}
In [40]:
#stratified kfold validation scheme:
skf = StratifiedKFold(n_splits=10, shuffle=True, random_state = 752)
In [41]:
# checking for correlations that can be explored for feature engineering;
plt.figure(figsize=(15,8))
sns.heatmap(train.iloc[:,1:train.shape[1]-1].corr(),annot=True, cmap='viridis');
Feature Engineering;¶
In [42]:
X = train.drop(['Entry_id','e_signed'], axis=1)
y = train.e_signed
# -----------------------------------------------------------------
X_pay_schedule = pd.get_dummies(X['pay_schedule']) #dummy variables;
X_copy = pd.concat((X,X_pay_schedule), axis=1)
X_copy.drop('pay_schedule',axis=1,inplace=True)
X = X_copy.copy()

# on test_dataset:
test_entry_Id = test['Entry_id']
test_pay_schedule = pd.get_dummies(test['pay_schedule'])
test_copy = pd.concat((test,test_pay_schedule), axis=1)
test_copy.drop('pay_schedule',axis=1,inplace=True)
test_copy.drop('Entry_id',axis=1,inplace=True)
In [43]:
'''discretizer = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile')

X['income_bins'] = discretizer.fit_transform(X['income'].values.reshape(-1,1).astype('float'))

X['amount_requested_bin'] = discretizer.fit_transform(X['amount_requested'].values.reshape(-1,1).astype('float'))

X['mean_risk_score'] = X[['risk_score_2','risk_score_3','risk_score_4',	'risk_score_5']].mean(axis=1)

X['mean_ext_quality'] = X[['ext_quality_score','ext_quality_score_2']].mean(axis=1)'''
Out[43]:
"discretizer = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='quantile')\n\nX['income_bins'] = discretizer.fit_transform(X['income'].values.reshape(-1,1).astype('float'))\n\nX['amount_requested_bin'] = discretizer.fit_transform(X['amount_requested'].values.reshape(-1,1).astype('float'))\n\nX['mean_risk_score'] = X[['risk_score_2','risk_score_3','risk_score_4',\t'risk_score_5']].mean(axis=1)\n\nX['mean_ext_quality'] = X[['ext_quality_score','ext_quality_score_2']].mean(axis=1)"
In [44]:
X.head()
Out[44]:
age home_owner income months_employed years_employed current_address_year personal_account_m personal_account_y has_debt amount_requested risk_score risk_score_2 risk_score_3 risk_score_4 risk_score_5 ext_quality_score ext_quality_score_2 inquiries_last_month bi-weekly monthly semi-monthly weekly
0 40 1 3135 0 3 3 6 2 1 550 36200 0.737398 0.903517 0.487712 0.515977 0.580918 0.380918 10 1 0 0 0
1 33 0 3590 0 5 2 2 8 1 1100 53850 0.617361 0.857560 0.613487 0.665523 0.744634 0.744634 12 0 0 1 0
2 21 0 2303 0 5 8 2 7 1 600 74850 0.677109 0.758765 0.495609 0.664762 0.592556 0.492556 6 0 0 0 1
3 26 0 2795 0 4 4 1 6 1 800 50800 0.738055 0.873204 0.666437 0.700392 0.584130 0.684130 14 1 0 0 0
4 43 0 5000 0 2 1 1 2 1 1100 69100 0.798303 0.841747 0.401971 0.568787 0.525905 0.725905 5 1 0 0 0
In [45]:
# in other to capture non-linear relationships, we compute the mutual_information_score of the features;
def make_mi_scores(X, y):
    X = X.copy()
    for colname in X.select_dtypes(["object", "category"]):
        X[colname], _ = X[colname].factorize()
    # All discrete features should now have integer dtypes
    discrete_features = [pd.api.types.is_integer_dtype(t) for t in X.dtypes]
    mi_scores = mutual_info_classif(X, y, discrete_features=discrete_features, random_state=0)
    mi_scores = pd.Series(mi_scores, name="MI Scores", index=X.columns)
    mi_scores = mi_scores.sort_values(ascending=False)
    return mi_scores


# to plot the score of Mutual Information 
def plot_mi_scores(scores):
    scores = scores.sort_values(ascending=True)
    width = np.arange(len(scores))
    ticks = list(scores.index)
    plt.barh(width, scores)
    plt.yticks(width, ticks)
    plt.title("Mutual Information Scores")
In [46]:
mi_scores = make_mi_scores(X,y)

plt.figure(figsize=(12,8))
plot_mi_scores(mi_scores)

from the plot above and the correlation as well as the EDA carried earlier, the features shows to hold information that can be exploited to make a robust model:

Model Building:¶

Training a LightGBM classifier:

In [47]:
lgb_clf = LGBMClassifier(random_state=752, class_weight=class_weight)
In [48]:
# Carrying Hyper Parameter Optuna Optimization using a Bayesian Optimizer;
def objective(trial):
    params ={
    'max_bin': trial.suggest_int('max_bin', 30, 150),
    'max_depth': trial.suggest_int('max_depth', 4, 15),
    'subsample': trial.suggest_uniform('subsample', 0.6,  1.0),
    'min_child_samples': trial.suggest_int('min_child_samples', 30, 100),
    'num_leaves': trial.suggest_int('num_leaves', 30, 100),
    'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.6, 0.9),
    'learning_rate': trial.suggest_loguniform('learning_rate', 0.1,  0.4),
    'bagging_freq': trial.suggest_int('bagging_freq', 6, 10),
    'lambda_l1': trial.suggest_uniform('lambda_l1', 0.0, 10.0),
    'lambda_l2': trial.suggest_uniform('lambda_l2', 0.0, 10.0),
    }
    lgb_clf.set_params(**params)

    
    
    return np.mean(cross_val_score(lgb_clf, X, y, cv=skf, scoring='roc_auc'))
In [49]:
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=15)
[I 2022-10-18 11:35:23,321] A new study created in memory with name: no-name-6d772233-9ca0-488e-a024-3198ee965290
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=3.660429140289728, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.660429140289728
[LightGBM] [Warning] lambda_l2 is set=3.6702986906413804, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.6702986906413804
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:35:35,354] Trial 0 finished with value: 0.6923767913269205 and parameters: {'max_bin': 41, 'max_depth': 10, 'subsample': 0.9867003879887545, 'min_child_samples': 64, 'num_leaves': 52, 'colsample_bytree': 0.7744347699848178, 'learning_rate': 0.1397854486575055, 'bagging_freq': 10, 'lambda_l1': 3.660429140289728, 'lambda_l2': 3.6702986906413804}. Best is trial 0 with value: 0.6923767913269205.
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=5.630950445534348, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.630950445534348
[LightGBM] [Warning] lambda_l2 is set=9.573659118627061, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.573659118627061
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[I 2022-10-18 11:35:42,647] Trial 1 finished with value: 0.6920266464095179 and parameters: {'max_bin': 64, 'max_depth': 12, 'subsample': 0.6240668724820215, 'min_child_samples': 32, 'num_leaves': 36, 'colsample_bytree': 0.6823320478190402, 'learning_rate': 0.11404079029070653, 'bagging_freq': 7, 'lambda_l1': 5.630950445534348, 'lambda_l2': 9.573659118627061}. Best is trial 0 with value: 0.6923767913269205.
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=9.830608328352218, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.830608328352218
[LightGBM] [Warning] lambda_l2 is set=6.0262929039209725, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.0262929039209725
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[I 2022-10-18 11:35:48,176] Trial 2 finished with value: 0.6788635560948342 and parameters: {'max_bin': 52, 'max_depth': 8, 'subsample': 0.644272973827263, 'min_child_samples': 35, 'num_leaves': 38, 'colsample_bytree': 0.7317253085581932, 'learning_rate': 0.23365141139363046, 'bagging_freq': 7, 'lambda_l1': 9.830608328352218, 'lambda_l2': 6.0262929039209725}. Best is trial 0 with value: 0.6923767913269205.
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=5.245279854663974, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.245279854663974
[LightGBM] [Warning] lambda_l2 is set=0.5023527141889284, reg_lambda=0.0 will be ignored. Current value: lambda_l2=0.5023527141889284
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:35:55,396] Trial 3 finished with value: 0.6645654913796282 and parameters: {'max_bin': 47, 'max_depth': 6, 'subsample': 0.6694009075830384, 'min_child_samples': 65, 'num_leaves': 58, 'colsample_bytree': 0.8391950621383191, 'learning_rate': 0.3705937101294885, 'bagging_freq': 10, 'lambda_l1': 5.245279854663974, 'lambda_l2': 0.5023527141889284}. Best is trial 0 with value: 0.6923767913269205.
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=1.0597424860446036, reg_alpha=0.0 will be ignored. Current value: lambda_l1=1.0597424860446036
[LightGBM] [Warning] lambda_l2 is set=6.124718037571873, reg_lambda=0.0 will be ignored. Current value: lambda_l2=6.124718037571873
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:36:02,412] Trial 4 finished with value: 0.6866143308433711 and parameters: {'max_bin': 34, 'max_depth': 9, 'subsample': 0.953024976776934, 'min_child_samples': 66, 'num_leaves': 44, 'colsample_bytree': 0.8518515800219988, 'learning_rate': 0.19077919398489906, 'bagging_freq': 10, 'lambda_l1': 1.0597424860446036, 'lambda_l2': 6.124718037571873}. Best is trial 0 with value: 0.6923767913269205.
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.4436495810824943, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.4436495810824943
[LightGBM] [Warning] lambda_l2 is set=9.620191328265546, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.620191328265546
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[I 2022-10-18 11:36:07,087] Trial 5 finished with value: 0.6925676992467031 and parameters: {'max_bin': 46, 'max_depth': 4, 'subsample': 0.8807866539784885, 'min_child_samples': 92, 'num_leaves': 55, 'colsample_bytree': 0.8123255435409104, 'learning_rate': 0.18938963311312057, 'bagging_freq': 8, 'lambda_l1': 2.4436495810824943, 'lambda_l2': 9.620191328265546}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=5.305921987044327, reg_alpha=0.0 will be ignored. Current value: lambda_l1=5.305921987044327
[LightGBM] [Warning] lambda_l2 is set=5.637039003473565, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.637039003473565
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:36:18,831] Trial 6 finished with value: 0.6890271632252188 and parameters: {'max_bin': 150, 'max_depth': 7, 'subsample': 0.7974702094624114, 'min_child_samples': 39, 'num_leaves': 55, 'colsample_bytree': 0.7513157163362637, 'learning_rate': 0.17796111509174836, 'bagging_freq': 9, 'lambda_l1': 5.305921987044327, 'lambda_l2': 5.637039003473565}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.2454749089624935, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.2454749089624935
[LightGBM] [Warning] lambda_l2 is set=2.4762278538629836, reg_lambda=0.0 will be ignored. Current value: lambda_l2=2.4762278538629836
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:36:29,452] Trial 7 finished with value: 0.6736581549147463 and parameters: {'max_bin': 118, 'max_depth': 10, 'subsample': 0.727814588517302, 'min_child_samples': 65, 'num_leaves': 46, 'colsample_bytree': 0.7295599177994592, 'learning_rate': 0.23133058101255896, 'bagging_freq': 9, 'lambda_l1': 3.2454749089624935, 'lambda_l2': 2.4762278538629836}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=8.796918562679933, reg_alpha=0.0 will be ignored. Current value: lambda_l1=8.796918562679933
[LightGBM] [Warning] lambda_l2 is set=5.524887265437544, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.524887265437544
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[I 2022-10-18 11:36:37,862] Trial 8 finished with value: 0.6724388928135637 and parameters: {'max_bin': 106, 'max_depth': 10, 'subsample': 0.6876908109461152, 'min_child_samples': 70, 'num_leaves': 81, 'colsample_bytree': 0.7182553772411413, 'learning_rate': 0.3042096036636965, 'bagging_freq': 8, 'lambda_l1': 8.796918562679933, 'lambda_l2': 5.524887265437544}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=7.026145177491992, reg_alpha=0.0 will be ignored. Current value: lambda_l1=7.026145177491992
[LightGBM] [Warning] lambda_l2 is set=5.077245578107769, reg_lambda=0.0 will be ignored. Current value: lambda_l2=5.077245578107769
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:36:47,953] Trial 9 finished with value: 0.6754705713359572 and parameters: {'max_bin': 92, 'max_depth': 11, 'subsample': 0.6063896987745414, 'min_child_samples': 83, 'num_leaves': 100, 'colsample_bytree': 0.6003576903709046, 'learning_rate': 0.16809156383596932, 'bagging_freq': 10, 'lambda_l1': 7.026145177491992, 'lambda_l2': 5.077245578107769}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[LightGBM] [Warning] lambda_l1 is set=0.8384812642637947, reg_alpha=0.0 will be ignored. Current value: lambda_l1=0.8384812642637947
[LightGBM] [Warning] lambda_l2 is set=9.744595189598138, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.744595189598138
[LightGBM] [Warning] bagging_freq is set=6, subsample_freq=0 will be ignored. Current value: bagging_freq=6
[I 2022-10-18 11:36:51,352] Trial 10 finished with value: 0.6892646704406881 and parameters: {'max_bin': 73, 'max_depth': 4, 'subsample': 0.8782146217256503, 'min_child_samples': 100, 'num_leaves': 73, 'colsample_bytree': 0.8888160473692124, 'learning_rate': 0.1211579740298769, 'bagging_freq': 6, 'lambda_l1': 0.8384812642637947, 'lambda_l2': 9.744595189598138}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=2.912468861268529, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.912468861268529
[LightGBM] [Warning] lambda_l2 is set=3.1137105006941197, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.1137105006941197
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[I 2022-10-18 11:37:00,532] Trial 11 finished with value: 0.6845451881228151 and parameters: {'max_bin': 36, 'max_depth': 14, 'subsample': 0.9559169658925827, 'min_child_samples': 52, 'num_leaves': 65, 'colsample_bytree': 0.7969361315190696, 'learning_rate': 0.13790387824101827, 'bagging_freq': 8, 'lambda_l1': 2.912468861268529, 'lambda_l2': 3.1137105006941197}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=3.3287283960951317, reg_alpha=0.0 will be ignored. Current value: lambda_l1=3.3287283960951317
[LightGBM] [Warning] lambda_l2 is set=7.892104596102517, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.892104596102517
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:37:06,427] Trial 12 finished with value: 0.6898962959894722 and parameters: {'max_bin': 72, 'max_depth': 4, 'subsample': 0.8579392807547597, 'min_child_samples': 98, 'num_leaves': 53, 'colsample_bytree': 0.7918252128170267, 'learning_rate': 0.14185491085824942, 'bagging_freq': 9, 'lambda_l1': 3.3287283960951317, 'lambda_l2': 7.892104596102517}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[LightGBM] [Warning] lambda_l1 is set=2.193507185715787, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.193507185715787
[LightGBM] [Warning] lambda_l2 is set=3.3954381303408425, reg_lambda=0.0 will be ignored. Current value: lambda_l2=3.3954381303408425
[LightGBM] [Warning] bagging_freq is set=7, subsample_freq=0 will be ignored. Current value: bagging_freq=7
[I 2022-10-18 11:37:23,456] Trial 13 finished with value: 0.6890021007038194 and parameters: {'max_bin': 55, 'max_depth': 15, 'subsample': 0.997790800861239, 'min_child_samples': 84, 'num_leaves': 76, 'colsample_bytree': 0.7941989904028371, 'learning_rate': 0.1524905127235007, 'bagging_freq': 7, 'lambda_l1': 2.193507185715787, 'lambda_l2': 3.3954381303408425}. Best is trial 5 with value: 0.6925676992467031.
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[LightGBM] [Warning] lambda_l1 is set=4.160651586352121, reg_alpha=0.0 will be ignored. Current value: lambda_l1=4.160651586352121
[LightGBM] [Warning] lambda_l2 is set=7.732030318358097, reg_lambda=0.0 will be ignored. Current value: lambda_l2=7.732030318358097
[LightGBM] [Warning] bagging_freq is set=8, subsample_freq=0 will be ignored. Current value: bagging_freq=8
[I 2022-10-18 11:37:29,673] Trial 14 finished with value: 0.6983593962984596 and parameters: {'max_bin': 83, 'max_depth': 13, 'subsample': 0.8971640076320018, 'min_child_samples': 51, 'num_leaves': 31, 'colsample_bytree': 0.6569151459124567, 'learning_rate': 0.10447652039740438, 'bagging_freq': 8, 'lambda_l1': 4.160651586352121, 'lambda_l2': 7.732030318358097}. Best is trial 14 with value: 0.6983593962984596.
In [50]:
study.best_value
Out[50]:
0.6983593962984596
In [51]:
lgb_clf.set_params(**study.best_params)
lgb_clf
# Parameters and Hyperparameters are taken note of for later:
Out[51]:
LGBMClassifier(bagging_freq=8,
               class_weight={0: 1.0883478260869566, 1: 0.9249187112030742},
               colsample_bytree=0.6569151459124567, lambda_l1=4.160651586352121,
               lambda_l2=7.732030318358097, learning_rate=0.10447652039740438,
               max_bin=83, max_depth=13, min_child_samples=51, random_state=752,
               subsample=0.8971640076320018)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
LGBMClassifier(bagging_freq=8,
               class_weight={0: 1.0883478260869566, 1: 0.9249187112030742},
               colsample_bytree=0.6569151459124567, lambda_l1=4.160651586352121,
               lambda_l2=7.732030318358097, learning_rate=0.10447652039740438,
               max_bin=83, max_depth=13, min_child_samples=51, random_state=752,
               subsample=0.8971640076320018)

Attempting a Narrower optimization:¶

In [52]:
from optuna import Trial, visualization
In [53]:
optuna.visualization.plot_optimization_history(study)
In [54]:
optuna.visualization.plot_slice(study)
In [55]:
# resetting Parameter range in other to improve lgbm score:
def objective(trial):
    params ={
    'max_bin': trial.suggest_int('max_bin', 45, 50),
    'max_depth': trial.suggest_int('max_depth', 6, 7),
    'subsample': trial.suggest_uniform('subsample', 0.79,0.81 ),
    'min_child_samples': trial.suggest_int('min_child_samples', 71, 75),
    'num_leaves': trial.suggest_int('num_leaves', 91, 93),
    'colsample_bytree': trial.suggest_uniform('colsample_bytree', 0.85, 0.9),
    'learning_rate': trial.suggest_loguniform('learning_rate', 0.15,  0.25),
    'bagging_freq': trial.suggest_int('bagging_freq', 9, 10),
    'lambda_l1': trial.suggest_uniform('lambda_l1', 2.8, 2.9),
    'lambda_l2': trial.suggest_uniform('lambda_l2', 8.15, 8.30),
    }
    lgb_clf.set_params(**params)

    
    
    return np.mean(cross_val_score(lgb_clf, X, y, cv=skf, scoring='roc_auc'))
In [56]:
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)
[I 2022-10-18 11:37:33,150] A new study created in memory with name: no-name-449440b7-c40f-41fc-8a64-7559ffbcbe23
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8199376144013204, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8199376144013204
[LightGBM] [Warning] lambda_l2 is set=8.23193978054748, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.23193978054748
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:37:42,277] Trial 0 finished with value: 0.6880573524823685 and parameters: {'max_bin': 48, 'max_depth': 7, 'subsample': 0.80746623030307, 'min_child_samples': 73, 'num_leaves': 92, 'colsample_bytree': 0.8764180684577638, 'learning_rate': 0.15565391169843945, 'bagging_freq': 10, 'lambda_l1': 2.8199376144013204, 'lambda_l2': 8.23193978054748}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8192130362366745, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8192130362366745
[LightGBM] [Warning] lambda_l2 is set=8.242163283033076, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.242163283033076
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:37:48,577] Trial 1 finished with value: 0.6852029554257859 and parameters: {'max_bin': 46, 'max_depth': 7, 'subsample': 0.7938988996922094, 'min_child_samples': 75, 'num_leaves': 92, 'colsample_bytree': 0.8553452921516038, 'learning_rate': 0.17230960921114752, 'bagging_freq': 9, 'lambda_l1': 2.8192130362366745, 'lambda_l2': 8.242163283033076}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8309181961284193, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8309181961284193
[LightGBM] [Warning] lambda_l2 is set=8.266196069847373, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.266196069847373
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:37:54,515] Trial 2 finished with value: 0.677312305267834 and parameters: {'max_bin': 47, 'max_depth': 7, 'subsample': 0.801796371124881, 'min_child_samples': 75, 'num_leaves': 92, 'colsample_bytree': 0.869616386185692, 'learning_rate': 0.24697537120185567, 'bagging_freq': 10, 'lambda_l1': 2.8309181961284193, 'lambda_l2': 8.266196069847373}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.884140511514055, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.884140511514055
[LightGBM] [Warning] lambda_l2 is set=8.271358055675366, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.271358055675366
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:03,174] Trial 3 finished with value: 0.6849672639054775 and parameters: {'max_bin': 48, 'max_depth': 6, 'subsample': 0.8065897390776072, 'min_child_samples': 71, 'num_leaves': 92, 'colsample_bytree': 0.8853656213372506, 'learning_rate': 0.22148377024769697, 'bagging_freq': 10, 'lambda_l1': 2.884140511514055, 'lambda_l2': 8.271358055675366}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8492144645553923, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8492144645553923
[LightGBM] [Warning] lambda_l2 is set=8.165710252546063, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.165710252546063
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:10,786] Trial 4 finished with value: 0.6874536053405302 and parameters: {'max_bin': 45, 'max_depth': 6, 'subsample': 0.80851900779079, 'min_child_samples': 74, 'num_leaves': 93, 'colsample_bytree': 0.8894566215517902, 'learning_rate': 0.2153786948049659, 'bagging_freq': 10, 'lambda_l1': 2.8492144645553923, 'lambda_l2': 8.165710252546063}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.828976141227842, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.828976141227842
[LightGBM] [Warning] lambda_l2 is set=8.277259756704213, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.277259756704213
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:18,119] Trial 5 finished with value: 0.684152520673578 and parameters: {'max_bin': 46, 'max_depth': 7, 'subsample': 0.8095985901349237, 'min_child_samples': 74, 'num_leaves': 92, 'colsample_bytree': 0.865616502545968, 'learning_rate': 0.2035411167140138, 'bagging_freq': 10, 'lambda_l1': 2.828976141227842, 'lambda_l2': 8.277259756704213}. Best is trial 0 with value: 0.6880573524823685.
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.853091289790148, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.853091289790148
[LightGBM] [Warning] lambda_l2 is set=8.168567118078128, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.168567118078128
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:23,010] Trial 6 finished with value: 0.688380891271477 and parameters: {'max_bin': 45, 'max_depth': 7, 'subsample': 0.8023017522055884, 'min_child_samples': 73, 'num_leaves': 93, 'colsample_bytree': 0.8866310397468276, 'learning_rate': 0.1973566744354138, 'bagging_freq': 9, 'lambda_l1': 2.853091289790148, 'lambda_l2': 8.168567118078128}. Best is trial 6 with value: 0.688380891271477.
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.840193979057465, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.840193979057465
[LightGBM] [Warning] lambda_l2 is set=8.278275701429344, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.278275701429344
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:26,042] Trial 7 finished with value: 0.6886381294463726 and parameters: {'max_bin': 50, 'max_depth': 6, 'subsample': 0.7988029977785731, 'min_child_samples': 72, 'num_leaves': 92, 'colsample_bytree': 0.8575932557699226, 'learning_rate': 0.21593418640809744, 'bagging_freq': 10, 'lambda_l1': 2.840193979057465, 'lambda_l2': 8.278275701429344}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8939458086829952, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8939458086829952
[LightGBM] [Warning] lambda_l2 is set=8.21145614878403, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.21145614878403
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:29,001] Trial 8 finished with value: 0.6816442981737785 and parameters: {'max_bin': 47, 'max_depth': 7, 'subsample': 0.795130645777246, 'min_child_samples': 72, 'num_leaves': 92, 'colsample_bytree': 0.8733514547921014, 'learning_rate': 0.17767881410078035, 'bagging_freq': 10, 'lambda_l1': 2.8939458086829952, 'lambda_l2': 8.21145614878403}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8259499953845957, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8259499953845957
[LightGBM] [Warning] lambda_l2 is set=8.220579979074472, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.220579979074472
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:31,421] Trial 9 finished with value: 0.685652357078608 and parameters: {'max_bin': 48, 'max_depth': 6, 'subsample': 0.8011825716341346, 'min_child_samples': 73, 'num_leaves': 91, 'colsample_bytree': 0.8605242355814204, 'learning_rate': 0.21393351059237856, 'bagging_freq': 10, 'lambda_l1': 2.8259499953845957, 'lambda_l2': 8.220579979074472}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.803638585269836, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.803638585269836
[LightGBM] [Warning] lambda_l2 is set=8.2974582407661, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.2974582407661
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:34,086] Trial 10 finished with value: 0.6840984343904368 and parameters: {'max_bin': 50, 'max_depth': 6, 'subsample': 0.7972164226975835, 'min_child_samples': 71, 'num_leaves': 91, 'colsample_bytree': 0.8510110523308755, 'learning_rate': 0.24979627963978987, 'bagging_freq': 9, 'lambda_l1': 2.803638585269836, 'lambda_l2': 8.2974582407661}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.86309137448722, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.86309137448722
[LightGBM] [Warning] lambda_l2 is set=8.155096411851895, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.155096411851895
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:36,968] Trial 11 finished with value: 0.6873317747793177 and parameters: {'max_bin': 50, 'max_depth': 6, 'subsample': 0.7908679658776684, 'min_child_samples': 72, 'num_leaves': 93, 'colsample_bytree': 0.8950800037023757, 'learning_rate': 0.19299178470027784, 'bagging_freq': 9, 'lambda_l1': 2.86309137448722, 'lambda_l2': 8.155096411851895}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8524300140401384, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8524300140401384
[LightGBM] [Warning] lambda_l2 is set=8.193485261440182, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.193485261440182
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:39,435] Trial 12 finished with value: 0.684151628406471 and parameters: {'max_bin': 49, 'max_depth': 6, 'subsample': 0.8036288366294362, 'min_child_samples': 72, 'num_leaves': 93, 'colsample_bytree': 0.88141263551768, 'learning_rate': 0.18995911555647926, 'bagging_freq': 9, 'lambda_l1': 2.8524300140401384, 'lambda_l2': 8.193485261440182}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8697045317857994, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8697045317857994
[LightGBM] [Warning] lambda_l2 is set=8.182589937247666, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.182589937247666
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:42,345] Trial 13 finished with value: 0.6804555650600665 and parameters: {'max_bin': 45, 'max_depth': 7, 'subsample': 0.7985737887269515, 'min_child_samples': 73, 'num_leaves': 93, 'colsample_bytree': 0.8928099513901816, 'learning_rate': 0.23017532756792153, 'bagging_freq': 9, 'lambda_l1': 2.8697045317857994, 'lambda_l2': 8.182589937247666}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8465784102632172, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8465784102632172
[LightGBM] [Warning] lambda_l2 is set=8.253290256918698, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.253290256918698
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:44,877] Trial 14 finished with value: 0.6869880186160054 and parameters: {'max_bin': 49, 'max_depth': 6, 'subsample': 0.8042118290157676, 'min_child_samples': 72, 'num_leaves': 91, 'colsample_bytree': 0.8801328894914282, 'learning_rate': 0.17958728503074806, 'bagging_freq': 9, 'lambda_l1': 2.8465784102632172, 'lambda_l2': 8.253290256918698}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.841032225713361, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.841032225713361
[LightGBM] [Warning] lambda_l2 is set=8.19879872691671, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.19879872691671
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:48,067] Trial 15 finished with value: 0.6842434372956251 and parameters: {'max_bin': 46, 'max_depth': 7, 'subsample': 0.7989476510979786, 'min_child_samples': 74, 'num_leaves': 93, 'colsample_bytree': 0.8987574530925121, 'learning_rate': 0.20257647893173467, 'bagging_freq': 9, 'lambda_l1': 2.841032225713361, 'lambda_l2': 8.19879872691671}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:50,581] Trial 16 finished with value: 0.6827979539418407 and parameters: {'max_bin': 49, 'max_depth': 6, 'subsample': 0.8045159719369179, 'min_child_samples': 71, 'num_leaves': 93, 'colsample_bytree': 0.8611406541829247, 'learning_rate': 0.23414709566642045, 'bagging_freq': 9, 'lambda_l1': 2.8675246457972445, 'lambda_l2': 8.29694944760606}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.8675246457972445, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8675246457972445
[LightGBM] [Warning] lambda_l2 is set=8.29694944760606, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.29694944760606
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.8794279764945654, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.8794279764945654
[LightGBM] [Warning] lambda_l2 is set=8.176487166737852, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.176487166737852
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:53,594] Trial 17 finished with value: 0.6870862166392614 and parameters: {'max_bin': 45, 'max_depth': 7, 'subsample': 0.7962203252309987, 'min_child_samples': 72, 'num_leaves': 91, 'colsample_bytree': 0.8875206881555803, 'learning_rate': 0.16375132647909488, 'bagging_freq': 10, 'lambda_l1': 2.8794279764945654, 'lambda_l2': 8.176487166737852}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[LightGBM] [Warning] lambda_l1 is set=2.859629121614426, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.859629121614426
[LightGBM] [Warning] lambda_l2 is set=8.25066429565717, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.25066429565717
[LightGBM] [Warning] bagging_freq is set=9, subsample_freq=0 will be ignored. Current value: bagging_freq=9
[I 2022-10-18 11:38:56,120] Trial 18 finished with value: 0.6849326623342817 and parameters: {'max_bin': 50, 'max_depth': 6, 'subsample': 0.7927592503550256, 'min_child_samples': 73, 'num_leaves': 92, 'colsample_bytree': 0.8502240454746174, 'learning_rate': 0.20471427996634178, 'bagging_freq': 9, 'lambda_l1': 2.859629121614426, 'lambda_l2': 8.25066429565717}. Best is trial 7 with value: 0.6886381294463726.
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[LightGBM] [Warning] lambda_l1 is set=2.838914252657792, reg_alpha=0.0 will be ignored. Current value: lambda_l1=2.838914252657792
[LightGBM] [Warning] lambda_l2 is set=8.152087452076726, reg_lambda=0.0 will be ignored. Current value: lambda_l2=8.152087452076726
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[I 2022-10-18 11:38:59,185] Trial 19 finished with value: 0.6800637537768397 and parameters: {'max_bin': 47, 'max_depth': 7, 'subsample': 0.8009514772946507, 'min_child_samples': 74, 'num_leaves': 93, 'colsample_bytree': 0.8688588908932673, 'learning_rate': 0.19210949065112495, 'bagging_freq': 10, 'lambda_l1': 2.838914252657792, 'lambda_l2': 8.152087452076726}. Best is trial 7 with value: 0.6886381294463726.
In [57]:
study.best_value
Out[57]:
0.6886381294463726

|

Training a XGBoost classifier:

In [58]:
def objective(trial):
    """Define the objective function"""

    params = {
        'max_depth': trial.suggest_int('max_depth', 1, 9),
        'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 1.0),
        'n_estimators': trial.suggest_int('n_estimators', 50, 500),
        'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
        'gamma': trial.suggest_loguniform('gamma', 1e-8, 1.0),
        'subsample': trial.suggest_loguniform('subsample', 0.01, 1.0),
        'colsample_bytree': trial.suggest_loguniform('colsample_bytree', 0.01, 1.0),
        'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-8, 1.0),
        'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-8, 1.0),
        'eval_metric': 'auc',
    }

    # Fit the model
    optuna_model = XGBClassifier(**params)
    return np.mean(cross_val_score(optuna_model, X, y, cv=skf, scoring='roc_auc'))
In [59]:
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)
[I 2022-10-18 11:38:59,269] A new study created in memory with name: no-name-70e645e7-73f9-4380-9d4e-a3861587f553
[I 2022-10-18 11:39:15,400] Trial 0 finished with value: 0.5178008916210874 and parameters: {'max_depth': 6, 'learning_rate': 0.9847490216038848, 'n_estimators': 477, 'min_child_weight': 10, 'gamma': 0.18821671584654576, 'subsample': 0.017709621134238157, 'colsample_bytree': 0.05464365512300051, 'reg_alpha': 1.5657202160942697e-07, 'reg_lambda': 0.17937833815937104}. Best is trial 0 with value: 0.5178008916210874.
[I 2022-10-18 11:39:48,479] Trial 1 finished with value: 0.6612398526467234 and parameters: {'max_depth': 6, 'learning_rate': 0.25103068566507947, 'n_estimators': 278, 'min_child_weight': 6, 'gamma': 9.84128192105661e-07, 'subsample': 0.7314351958369125, 'colsample_bytree': 0.6306191865534457, 'reg_alpha': 3.991828023826425e-06, 'reg_lambda': 1.1153461072066687e-05}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:39:51,727] Trial 2 finished with value: 0.5990593528631399 and parameters: {'max_depth': 6, 'learning_rate': 0.03827346388240056, 'n_estimators': 60, 'min_child_weight': 7, 'gamma': 7.74429837346125e-05, 'subsample': 0.01930919569706319, 'colsample_bytree': 0.3388439116891192, 'reg_alpha': 4.630891212960109e-05, 'reg_lambda': 0.8112767987745361}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:40:07,047] Trial 3 finished with value: 0.6419780055018072 and parameters: {'max_depth': 1, 'learning_rate': 0.28881014454429016, 'n_estimators': 413, 'min_child_weight': 2, 'gamma': 0.00042585128589447507, 'subsample': 0.8223426188644173, 'colsample_bytree': 0.010208820571352164, 'reg_alpha': 0.0903601102222745, 'reg_lambda': 0.00019336530202732034}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:40:10,850] Trial 4 finished with value: 0.6283895771953552 and parameters: {'max_depth': 1, 'learning_rate': 0.081409110645201, 'n_estimators': 89, 'min_child_weight': 6, 'gamma': 0.005557502980415191, 'subsample': 0.20250095878084842, 'colsample_bytree': 0.8754646135563454, 'reg_alpha': 2.474608752436114e-08, 'reg_lambda': 0.05430958068368139}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:40:17,814] Trial 5 finished with value: 0.5891206692459316 and parameters: {'max_depth': 9, 'learning_rate': 0.8899459220198809, 'n_estimators': 84, 'min_child_weight': 1, 'gamma': 5.702187455046356e-08, 'subsample': 0.5269809660159449, 'colsample_bytree': 0.05726775253400361, 'reg_alpha': 0.10138215804709787, 'reg_lambda': 0.002795325518412142}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:40:26,222] Trial 6 finished with value: 0.6544533530128646 and parameters: {'max_depth': 5, 'learning_rate': 0.102321601366619, 'n_estimators': 129, 'min_child_weight': 10, 'gamma': 1.7683576777288827e-07, 'subsample': 0.201979996250591, 'colsample_bytree': 0.4312823076999437, 'reg_alpha': 0.018043829123886995, 'reg_lambda': 1.9869503084334187e-07}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:40:41,856] Trial 7 finished with value: 0.6412458169279182 and parameters: {'max_depth': 1, 'learning_rate': 0.1964699114980169, 'n_estimators': 402, 'min_child_weight': 2, 'gamma': 0.005928224586160517, 'subsample': 0.19857881835084615, 'colsample_bytree': 0.8131446819830787, 'reg_alpha': 0.006204789337660842, 'reg_lambda': 4.970815997187347e-06}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:41:03,332] Trial 8 finished with value: 0.6021807473611048 and parameters: {'max_depth': 7, 'learning_rate': 0.0819790492734755, 'n_estimators': 493, 'min_child_weight': 5, 'gamma': 4.576593169032308e-07, 'subsample': 0.01716524634551705, 'colsample_bytree': 0.09619909608772118, 'reg_alpha': 2.917828766375959e-07, 'reg_lambda': 0.0006742803840984148}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:41:19,449] Trial 9 finished with value: 0.6574118302914043 and parameters: {'max_depth': 7, 'learning_rate': 0.010058179437484176, 'n_estimators': 252, 'min_child_weight': 6, 'gamma': 1.1531844307491352e-07, 'subsample': 0.14489241080906876, 'colsample_bytree': 0.187548347486818, 'reg_alpha': 0.0025698296442670573, 'reg_lambda': 5.168754043135982e-08}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:41:29,461] Trial 10 finished with value: 0.5960521983857869 and parameters: {'max_depth': 4, 'learning_rate': 0.31463175967565193, 'n_estimators': 249, 'min_child_weight': 8, 'gamma': 1.0566432194790471e-05, 'subsample': 0.051328673278367536, 'colsample_bytree': 0.015956442986133683, 'reg_alpha': 7.696621311614376e-06, 'reg_lambda': 3.5207519627574413e-06}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:41:45,847] Trial 11 finished with value: 0.6510752156850524 and parameters: {'max_depth': 8, 'learning_rate': 0.010789786876001042, 'n_estimators': 264, 'min_child_weight': 4, 'gamma': 2.1201549844205974e-06, 'subsample': 0.07057608079134034, 'colsample_bytree': 0.2778573937185178, 'reg_alpha': 0.0008657176380372058, 'reg_lambda': 1.4183270067137154e-08}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:41:58,243] Trial 12 finished with value: 0.6609970847414344 and parameters: {'max_depth': 4, 'learning_rate': 0.011285665012928155, 'n_estimators': 198, 'min_child_weight': 4, 'gamma': 1.9313745928312004e-08, 'subsample': 0.44272987595232044, 'colsample_bytree': 0.17553297652429733, 'reg_alpha': 6.876706283625655e-06, 'reg_lambda': 4.9345075838917885e-06}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:42:09,029] Trial 13 finished with value: 0.6597928321462405 and parameters: {'max_depth': 3, 'learning_rate': 0.02934881962699279, 'n_estimators': 173, 'min_child_weight': 4, 'gamma': 1.8003996810940786e-08, 'subsample': 0.46432816895706447, 'colsample_bytree': 0.14718728876288084, 'reg_alpha': 1.5205060809970712e-05, 'reg_lambda': 5.739139801030376e-06}. Best is trial 1 with value: 0.6612398526467234.
[I 2022-10-18 11:42:37,070] Trial 14 finished with value: 0.6862289485954585 and parameters: {'max_depth': 3, 'learning_rate': 0.02631881316847292, 'n_estimators': 336, 'min_child_weight': 4, 'gamma': 1.0131566126567182e-08, 'subsample': 0.9646430480076363, 'colsample_bytree': 0.722218718616715, 'reg_alpha': 3.273715718793605e-06, 'reg_lambda': 3.965548535252624e-05}. Best is trial 14 with value: 0.6862289485954585.
[I 2022-10-18 11:43:05,140] Trial 15 finished with value: 0.6845960382276413 and parameters: {'max_depth': 3, 'learning_rate': 0.02464792616376107, 'n_estimators': 351, 'min_child_weight': 3, 'gamma': 2.724781510113401e-06, 'subsample': 0.8746563005909297, 'colsample_bytree': 0.5062718319817441, 'reg_alpha': 7.26294041746106e-07, 'reg_lambda': 6.739938550097768e-05}. Best is trial 14 with value: 0.6862289485954585.
[I 2022-10-18 11:43:27,614] Trial 16 finished with value: 0.6794706260576197 and parameters: {'max_depth': 3, 'learning_rate': 0.02561348388343415, 'n_estimators': 354, 'min_child_weight': 3, 'gamma': 1.5010709559835583e-05, 'subsample': 0.3385866854685991, 'colsample_bytree': 0.49290383702988505, 'reg_alpha': 9.097100486995425e-07, 'reg_lambda': 0.009077419624687784}. Best is trial 14 with value: 0.6862289485954585.
[I 2022-10-18 11:43:55,511] Trial 17 finished with value: 0.6867603909406768 and parameters: {'max_depth': 3, 'learning_rate': 0.019842835945463603, 'n_estimators': 342, 'min_child_weight': 3, 'gamma': 6.908021108956782e-06, 'subsample': 0.9098830935537209, 'colsample_bytree': 0.9985723966973875, 'reg_alpha': 0.00018406912928973625, 'reg_lambda': 0.00010144407751914216}. Best is trial 17 with value: 0.6867603909406768.
[I 2022-10-18 11:44:08,644] Trial 18 finished with value: 0.6341057287120382 and parameters: {'max_depth': 2, 'learning_rate': 0.04661776990751693, 'n_estimators': 327, 'min_child_weight': 1, 'gamma': 0.0003663235447394514, 'subsample': 0.03159408018214182, 'colsample_bytree': 0.8711452112999739, 'reg_alpha': 0.00034831506109148314, 'reg_lambda': 0.00015036298378285723}. Best is trial 17 with value: 0.6867603909406768.
[I 2022-10-18 11:44:27,979] Trial 19 finished with value: 0.6477454804276954 and parameters: {'max_depth': 4, 'learning_rate': 0.016320379772142894, 'n_estimators': 420, 'min_child_weight': 8, 'gamma': 0.2123624284680917, 'subsample': 0.2767432085429712, 'colsample_bytree': 0.028340793651780348, 'reg_alpha': 0.0001400554529589211, 'reg_lambda': 4.50988560749194e-07}. Best is trial 17 with value: 0.6867603909406768.
In [60]:
study.best_value
Out[60]:
0.6867603909406768
In [61]:
study.best_params
Out[61]:
{'max_depth': 3,
 'learning_rate': 0.019842835945463603,
 'n_estimators': 342,
 'min_child_weight': 3,
 'gamma': 6.908021108956782e-06,
 'subsample': 0.9098830935537209,
 'colsample_bytree': 0.9985723966973875,
 'reg_alpha': 0.00018406912928973625,
 'reg_lambda': 0.00010144407751914216}
In [ ]:
 

Model Ensembling:¶

model 1:

In [62]:
lgbm_clf = LGBMClassifier(bagging_freq=10,
               class_weight={0: 1.0883478260869566, 1: 0.9249187112030742},
               colsample_bytree=0.6869647898646877, lambda_l1=9.543658512845797,
               lambda_l2=9.999550128682136, learning_rate=0.2637289335936835,
               max_bin=150, max_depth=4, metric='auc', min_child_samples=57,
               n_estimators=4000, num_leaves=67, random_state=752,
               subsample=0.8886905347670871)
In [63]:
# Flexible Stratified K-Fold Cross Validation

def lgb_predict(estimator, train, label, test, estimator_name): 
    mean_train = [] 
    mean_test_val = [] 
    test_pred = [] 
    val_pred = np.empty((len(train))) 
    for count, (train_index,test_index) in enumerate(skf.split(train,label)):
        x_train,x_test = train.iloc[train_index], train.iloc[test_index] 
        y_train,y_test = label.iloc[train_index], label.iloc[test_index] 
        
        print(f'========================Fold{count +1}==========================')
        
        
        estimator.fit(x_train, y_train, eval_set = [(x_train, y_train), (x_test, y_test)], verbose = 100,
                     early_stopping_rounds=200, eval_metric='auc'
                     )
                     
        
        train_predict = estimator.predict_proba(x_train)[:, 1]
        test_predict = estimator.predict_proba(x_test)[:, 1]
        val_pred[test_index] = (test_predict) # housing the lgbm-meta train data for stacking
        f_test= estimator.predict_proba(test)[:, 1]
        test_pred.append(f_test) # housing the lgbm-meta-test data for stacking

        print('\nValidation scores', roc_auc_score(y_test, test_predict))
        print('\nTraining scores', roc_auc_score(y_train, train_predict))
        mean_train.append(roc_auc_score(y_train, train_predict))
        mean_test_val.append(roc_auc_score(y_test,test_predict))
    print('Average Testing AUC_score score for 10 folds split:',np.mean(mean_test_val))
    print('Average Training AUC_score score for 10 folds split:',np.mean(mean_train))
    print('standard Deviation for 10 folds split:',np.std(mean_test_val))
    return val_pred, test_pred, estimator_name # return the meta train, meta-test, and estimator name
In [64]:
lgb_train, lgb_test, lgb_estimator = lgb_predict(lgbm_clf, X, y, test_copy, 'lgb_clf')
========================Fold1==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.795527	valid_1's auc: 0.698254
[200]	training's auc: 0.845132	valid_1's auc: 0.686905
[300]	training's auc: 0.875351	valid_1's auc: 0.686987

Validation scores 0.6995928328302614

Training scores 0.7975803043898755
========================Fold2==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.792974	valid_1's auc: 0.690114
[200]	training's auc: 0.845577	valid_1's auc: 0.687195

Validation scores 0.6949277503050544

Training scores 0.7813453420002651
========================Fold3==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.79339	valid_1's auc: 0.684645
[200]	training's auc: 0.840974	valid_1's auc: 0.678988
[300]	training's auc: 0.873862	valid_1's auc: 0.677018

Validation scores 0.6862475113993964

Training scores 0.7938954144759339
========================Fold4==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.796506	valid_1's auc: 0.684986
[200]	training's auc: 0.846051	valid_1's auc: 0.688467
[300]	training's auc: 0.874559	valid_1's auc: 0.685836

Validation scores 0.6889782287585897

Training scores 0.8449914671503137
========================Fold5==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.79433	valid_1's auc: 0.715263
[200]	training's auc: 0.84022	valid_1's auc: 0.715181
[300]	training's auc: 0.874294	valid_1's auc: 0.713521

Validation scores 0.7182043542482821

Training scores 0.812473272861571
========================Fold6==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.792621	valid_1's auc: 0.674143
[200]	training's auc: 0.840345	valid_1's auc: 0.681634
[300]	training's auc: 0.87485	valid_1's auc: 0.6761
[400]	training's auc: 0.896765	valid_1's auc: 0.668479

Validation scores 0.6826510821398755

Training scores 0.8419625157585985
========================Fold7==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.794444	valid_1's auc: 0.677067
[200]	training's auc: 0.845703	valid_1's auc: 0.679756
[300]	training's auc: 0.876694	valid_1's auc: 0.676743

Validation scores 0.6810470800102908

Training scores 0.838303419718712
========================Fold8==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.793997	valid_1's auc: 0.701716
[200]	training's auc: 0.842673	valid_1's auc: 0.700916

Validation scores 0.7087419603807563

Training scores 0.7683402108469575
========================Fold9==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.789256	valid_1's auc: 0.699359
[200]	training's auc: 0.841032	valid_1's auc: 0.689789

Validation scores 0.7080061744275791

Training scores 0.7596681659170414
========================Fold10==========================
[LightGBM] [Warning] lambda_l1 is set=9.543658512845797, reg_alpha=0.0 will be ignored. Current value: lambda_l1=9.543658512845797
[LightGBM] [Warning] lambda_l2 is set=9.999550128682136, reg_lambda=0.0 will be ignored. Current value: lambda_l2=9.999550128682136
[LightGBM] [Warning] bagging_freq is set=10, subsample_freq=0 will be ignored. Current value: bagging_freq=10
[100]	training's auc: 0.792941	valid_1's auc: 0.68353
[200]	training's auc: 0.843641	valid_1's auc: 0.680286
[300]	training's auc: 0.875357	valid_1's auc: 0.679892

Validation scores 0.684998713660921

Training scores 0.8151549622014389
Average Testing AUC_score score for 10 folds split: 0.6953395688161007
Average Training AUC_score score for 10 folds split: 0.8053715075320707
standard Deviation for 10 folds split: 0.012150981979339247
In [65]:
lgb_test_ = np.mean(lgb_test, axis=0) 
In [66]:
plt.figure(figsize=(12,8))
feat_importances = pd.Series(lgbm_clf.feature_importances_, index=X.columns)
feat_importances.sort_values().plot(kind='barh',title='lightgbm_1 feature importance');

model_2:

In [67]:
xgb_clf = XGBClassifier(colsample_bytree=0.6682960151866543, gamma=1.0798626152988356e-08,
              learning_rate=0.02320865571556698, max_depth=6,
              min_child_weight=3, n_estimators=196,
              reg_alpha=0.060016599138875734, reg_lambda=3.429258701297801e-05,
              subsample=0.5054109001304139)
In [68]:
def xgb_predict(estimator, train, label, test, estimator_name):
    mean_train = []
    mean_test_val = []
    test_pred = []
    val_pred = np.empty((len(train)))
    for count, (train_index,test_index) in enumerate(skf.split(train,label)):
        x_train,x_test = train.iloc[train_index], train.iloc[test_index]
        y_train,y_test = label.iloc[train_index], label.iloc[test_index]
        
        print(f'========================Fold{count +1}==========================')
        
        
        estimator.fit(x_train, y_train, eval_set = [(x_train, y_train), (x_test, y_test)], verbose = 100,
                     early_stopping_rounds=200 
                     )
                     
        
        train_predict = estimator.predict_proba(x_train)[:, 1]
        test_predict = estimator.predict_proba(x_test)[:, 1]
        val_pred[test_index] = (test_predict)
        f_test= estimator.predict_proba(test)[:, 1]
        test_pred.append(f_test)
    
        print('\nValidation scores', roc_auc_score(y_test, test_predict))
        print('\nTraining scores', roc_auc_score(y_train, train_predict))
        mean_train.append(roc_auc_score(y_train, train_predict))
        mean_test_val.append(roc_auc_score(y_test,test_predict))
    print('Average Testing AUC_score score for 10 folds split:',np.mean(mean_test_val))
    print('Average Training AUC_score score for 10 folds split:',np.mean(mean_train))
    print('standard Deviation for 10 folds split:',np.std(mean_test_val))
    return val_pred, test_pred, estimator_name
In [69]:
xgb_train, xgb_test, xgb_estimator = xgb_predict(xgb_clf, X, y, test_copy, 'xgb_clf')
========================Fold1==========================
[0]	validation_0-logloss:0.69099	validation_1-logloss:0.69143
[100]	validation_0-logloss:0.60411	validation_1-logloss:0.64130
[195]	validation_0-logloss:0.56846	validation_1-logloss:0.63094

Validation scores 0.6972628604457003

Training scores 0.8210335895171701
========================Fold2==========================
[0]	validation_0-logloss:0.69115	validation_1-logloss:0.69138
[100]	validation_0-logloss:0.60376	validation_1-logloss:0.63566
[195]	validation_0-logloss:0.56676	validation_1-logloss:0.62794

Validation scores 0.692466765140325

Training scores 0.8253271798435922
========================Fold3==========================
[0]	validation_0-logloss:0.69119	validation_1-logloss:0.69152
[100]	validation_0-logloss:0.60084	validation_1-logloss:0.63842
[195]	validation_0-logloss:0.56558	validation_1-logloss:0.62824

Validation scores 0.6855898786205125

Training scores 0.8228674817898436
========================Fold4==========================
[0]	validation_0-logloss:0.69104	validation_1-logloss:0.69203
[100]	validation_0-logloss:0.60212	validation_1-logloss:0.64569
[195]	validation_0-logloss:0.56599	validation_1-logloss:0.63963

Validation scores 0.6771125810802133

Training scores 0.8221102598096036
========================Fold5==========================
[0]	validation_0-logloss:0.69103	validation_1-logloss:0.69080
[100]	validation_0-logloss:0.60373	validation_1-logloss:0.62277
[195]	validation_0-logloss:0.56818	validation_1-logloss:0.60859

Validation scores 0.7266713762764113

Training scores 0.8254295264367598
========================Fold6==========================
[0]	validation_0-logloss:0.69107	validation_1-logloss:0.69173
[100]	validation_0-logloss:0.60284	validation_1-logloss:0.64220
[195]	validation_0-logloss:0.56671	validation_1-logloss:0.63186

Validation scores 0.6902061524629118

Training scores 0.8252930801802252
========================Fold7==========================
[0]	validation_0-logloss:0.69114	validation_1-logloss:0.69131
[100]	validation_0-logloss:0.60131	validation_1-logloss:0.63836
[195]	validation_0-logloss:0.56792	validation_1-logloss:0.63035

Validation scores 0.6853820427064574

Training scores 0.8188160681563981
========================Fold8==========================
[0]	validation_0-logloss:0.69113	validation_1-logloss:0.69127
[100]	validation_0-logloss:0.60195	validation_1-logloss:0.62701
[195]	validation_0-logloss:0.56602	validation_1-logloss:0.62005

Validation scores 0.7012271674813481

Training scores 0.8245020981572706
========================Fold9==========================
[0]	validation_0-logloss:0.69116	validation_1-logloss:0.69120
[100]	validation_0-logloss:0.60123	validation_1-logloss:0.62928
[195]	validation_0-logloss:0.56833	validation_1-logloss:0.62097

Validation scores 0.7085644455878569

Training scores 0.8189070068140534
========================Fold10==========================
[0]	validation_0-logloss:0.69102	validation_1-logloss:0.69134
[100]	validation_0-logloss:0.60055	validation_1-logloss:0.63466
[195]	validation_0-logloss:0.56642	validation_1-logloss:0.63097

Validation scores 0.693293028042192

Training scores 0.8086236246955887
Average Testing AUC_score score for 10 folds split: 0.695777629784393
Average Training AUC_score score for 10 folds split: 0.8212909915400506
standard Deviation for 10 folds split: 0.013259896604372852
In [70]:
xgb_test_ = np.mean(xgb_test, axis=0)

model_3:

In [71]:
xgb_2 = XGBClassifier()
xgb_2.set_params(**{'max_depth': 8,
 'learning_rate': 0.03097330142137865,
 'n_estimators': 499,
 'min_child_weight': 3,
 'gamma': 0.00014108851650059825,
 'subsample': 0.9231145138691651,
 'colsample_bytree': 0.9670536338693626,
 'reg_alpha': 0.00011310709950064508,
 'reg_lambda': 1.2597877821986132e-08})
Out[71]:
XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=0.9670536338693626, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None,
              gamma=0.00014108851650059825, gpu_id=None, grow_policy=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=0.03097330142137865, max_bin=None,
              max_cat_to_onehot=None, max_delta_step=None, max_depth=8,
              max_leaves=None, min_child_weight=3, missing=nan,
              monotone_constraints=None, n_estimators=499, n_jobs=None,
              num_parallel_tree=None, predictor=None, random_state=None,
              reg_alpha=0.00011310709950064508,
              reg_lambda=1.2597877821986132e-08, ...)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
XGBClassifier(base_score=None, booster=None, callbacks=None,
              colsample_bylevel=None, colsample_bynode=None,
              colsample_bytree=0.9670536338693626, early_stopping_rounds=None,
              enable_categorical=False, eval_metric=None,
              gamma=0.00014108851650059825, gpu_id=None, grow_policy=None,
              importance_type=None, interaction_constraints=None,
              learning_rate=0.03097330142137865, max_bin=None,
              max_cat_to_onehot=None, max_delta_step=None, max_depth=8,
              max_leaves=None, min_child_weight=3, missing=nan,
              monotone_constraints=None, n_estimators=499, n_jobs=None,
              num_parallel_tree=None, predictor=None, random_state=None,
              reg_alpha=0.00011310709950064508,
              reg_lambda=1.2597877821986132e-08, ...)
In [72]:
xgb_2_train, xgb_2_test, xgb_2_estimator = xgb_predict(xgb_2, X, y, test_copy, 'xgb_clf_2')
========================Fold1==========================
[0]	validation_0-logloss:0.68923	validation_1-logloss:0.69038
[100]	validation_0-logloss:0.52919	validation_1-logloss:0.62294
[200]	validation_0-logloss:0.45887	validation_1-logloss:0.61575
[300]	validation_0-logloss:0.40953	validation_1-logloss:0.61404
[400]	validation_0-logloss:0.36457	validation_1-logloss:0.61301
[498]	validation_0-logloss:0.32806	validation_1-logloss:0.61516

Validation scores 0.7182762828334724

Training scores 0.9888849219666731
========================Fold2==========================
[0]	validation_0-logloss:0.68897	validation_1-logloss:0.69029
[100]	validation_0-logloss:0.52761	validation_1-logloss:0.61902
[200]	validation_0-logloss:0.46309	validation_1-logloss:0.61631
[300]	validation_0-logloss:0.41278	validation_1-logloss:0.61918
[400]	validation_0-logloss:0.36929	validation_1-logloss:0.62171

Validation scores 0.7003789095112709

Training scores 0.9385252728647446
========================Fold3==========================
[0]	validation_0-logloss:0.68886	validation_1-logloss:0.69067
[100]	validation_0-logloss:0.52480	validation_1-logloss:0.62546
[200]	validation_0-logloss:0.46502	validation_1-logloss:0.62169
[300]	validation_0-logloss:0.41325	validation_1-logloss:0.62093
[400]	validation_0-logloss:0.36823	validation_1-logloss:0.62414
[443]	validation_0-logloss:0.35009	validation_1-logloss:0.62611

Validation scores 0.6952668422066662

Training scores 0.9496416044454916
========================Fold4==========================
[0]	validation_0-logloss:0.68909	validation_1-logloss:0.69113
[100]	validation_0-logloss:0.52784	validation_1-logloss:0.63756
[200]	validation_0-logloss:0.46635	validation_1-logloss:0.63621
[300]	validation_0-logloss:0.41508	validation_1-logloss:0.63527
[400]	validation_0-logloss:0.37123	validation_1-logloss:0.63540
[498]	validation_0-logloss:0.33632	validation_1-logloss:0.63798

Validation scores 0.6864632971549676

Training scores 0.9762135092742673
========================Fold5==========================
[0]	validation_0-logloss:0.68892	validation_1-logloss:0.68946
[100]	validation_0-logloss:0.53275	validation_1-logloss:0.60256
[200]	validation_0-logloss:0.46860	validation_1-logloss:0.59920
[300]	validation_0-logloss:0.41486	validation_1-logloss:0.59874
[400]	validation_0-logloss:0.37130	validation_1-logloss:0.60027
[442]	validation_0-logloss:0.35379	validation_1-logloss:0.60001

Validation scores 0.7308971806563485

Training scores 0.9517173837671958
========================Fold6==========================
[0]	validation_0-logloss:0.68914	validation_1-logloss:0.69100
[100]	validation_0-logloss:0.52565	validation_1-logloss:0.62840
[200]	validation_0-logloss:0.46306	validation_1-logloss:0.62679
[300]	validation_0-logloss:0.41116	validation_1-logloss:0.62849
[336]	validation_0-logloss:0.39669	validation_1-logloss:0.62813

Validation scores 0.6945655385010598

Training scores 0.9027327333760173
========================Fold7==========================
[0]	validation_0-logloss:0.68900	validation_1-logloss:0.69063
[100]	validation_0-logloss:0.52805	validation_1-logloss:0.62535
[200]	validation_0-logloss:0.46855	validation_1-logloss:0.62328
[300]	validation_0-logloss:0.41605	validation_1-logloss:0.62643
[367]	validation_0-logloss:0.38696	validation_1-logloss:0.62789

Validation scores 0.6935914587085156

Training scores 0.9180671093024917
========================Fold8==========================
[0]	validation_0-logloss:0.68937	validation_1-logloss:0.69025
[100]	validation_0-logloss:0.53095	validation_1-logloss:0.60981
[200]	validation_0-logloss:0.46755	validation_1-logloss:0.60668
[300]	validation_0-logloss:0.41416	validation_1-logloss:0.60672
[400]	validation_0-logloss:0.37248	validation_1-logloss:0.60449
[498]	validation_0-logloss:0.33494	validation_1-logloss:0.60663

Validation scores 0.7174324671983534

Training scores 0.9863795086583692
========================Fold9==========================
[0]	validation_0-logloss:0.68918	validation_1-logloss:0.69045
[100]	validation_0-logloss:0.52991	validation_1-logloss:0.61273
[200]	validation_0-logloss:0.47035	validation_1-logloss:0.60818
[300]	validation_0-logloss:0.41911	validation_1-logloss:0.60772
[400]	validation_0-logloss:0.37545	validation_1-logloss:0.60829
[498]	validation_0-logloss:0.33843	validation_1-logloss:0.60950

Validation scores 0.7221456135837406

Training scores 0.9690082260457074
========================Fold10==========================
[0]	validation_0-logloss:0.68910	validation_1-logloss:0.69005
[100]	validation_0-logloss:0.53147	validation_1-logloss:0.62495
[200]	validation_0-logloss:0.46190	validation_1-logloss:0.62515
[300]	validation_0-logloss:0.41396	validation_1-logloss:0.62844
[383]	validation_0-logloss:0.37733	validation_1-logloss:0.63279

Validation scores 0.6997478775405197

Training scores 0.9300107565264988
Average Testing AUC_score score for 10 folds split: 0.7058765467894915
Average Training AUC_score score for 10 folds split: 0.9511181026227458
standard Deviation for 10 folds split: 0.01419187935645962
In [73]:
xgb_2_test = np.mean(xgb_2_test, axis=0)
In [74]:
# preparing model results for ensembling:
xgb_train = pd.DataFrame(xgb_train)
xgb_2_train = pd.DataFrame(xgb_2_train)
lgb_train = pd.DataFrame(lgb_train)
# ---------------------------------------
stack_train = pd.concat((xgb_train, xgb_2_train, lgb_train), axis=1)
stack_train.head()
Out[74]:
0 0 0
0 0.576665 0.565816 0.737682
1 0.427056 0.427855 0.206949
2 0.700967 0.700313 0.650142
3 0.585420 0.558738 0.454794
4 0.556723 0.482709 0.451343
In [75]:
xgb_test_ = pd.DataFrame(xgb_test_)
xgb_2_test = pd.DataFrame(xgb_2_test)
lgb_test_ = pd.DataFrame(lgb_test_)
# -----------------------------------
stack_test = pd.concat((xgb_test_, xgb_2_test, lgb_test_), axis=1)
stack_test.head()
Out[75]:
0 0 0
0 0.360520 0.308466 0.333925
1 0.720678 0.840729 0.776093
2 0.530376 0.587887 0.517935
3 0.571721 0.576584 0.611566
4 0.393138 0.332143 0.255641
In [76]:
# using Linear Regression as the first meta model 
lr = LinearRegression()
lr.fit(stack_train, y)

pred = lr.predict(stack_train)
pred
Out[76]:
array([0.62509204, 0.36670404, 0.70218927, ..., 0.44126959, 0.53779044,
       0.65739997])
In [77]:
print(roc_auc_score(y, pred))
0.7081958128237094
In [78]:
# using Logistic Regression as the second meta model:
lg = LogisticRegression()
lg.fit(stack_train,y)

lg_pred = lg.predict_proba(stack_train)
In [79]:
print(roc_auc_score(y, lg_pred[:,1]))
0.708093948000874
In [80]:
# on stack test;

pred = lr.predict(stack_test)

pred_lg = lg.predict_proba(stack_test)[:,1]

preparing for submission:`¶

In [81]:
stack_submission_dict = {
    'Entry_id': test_entry_Id,
    'e_signed' : pred_lg #logisticRegression meta model performed better on LB
}
In [82]:
stack_submission = pd.DataFrame(stack_submission_dict)
#stack_submission.to_csv('stacked_models_lg.csv',index=False)
In [ ]: